The approaches described in this background section are not necessarily prior art to the claims in this application and are not admitted as prior art by inclusion in this section.
Application performance monitoring can be of two types—passive and synthetic. In passive monitoring, real-world historical performance of an application (e.g., an application hosted on a server such as a web server or an application server) is typically assessed based on monitoring actual log-ins, site hits, clicks, requests for data and other server transactions.
Synthetic monitoring, on the other hand, is performed by periodically deploying scripted tests that simulate user interactions with an application. A log of the transaction tests is generally saved and administrators can then review the test log to determine whether a test was successful or not. Synthetic monitoring is useful as it allows administrators to test new applications before they are launched, measure the way end users interact with applications (such as websites, APIs or mobile apps), and quickly identify faults. Synthetic monitoring can also provide direct feedback on performance degradation or availability issues allowing administrators to locate the root cause of failures and fix those issues before they impact end users.
As test scripts are periodically deployed in synthetic monitoring to check performance, this type of monitoring is often automated—i.e., programs are utilized to automatically run the tests at predetermined times. One issue that particularly arises because of this automation is that it is difficult to determine whether a particular monitoring test failed because of a fault with the application it was testing or because of a fault with the test itself. This is often referred to as the problem of ‘flakey’ tests.